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ABSTRACT 

This monograph provides definitions, guidelines, and 
recommended practices for assessing and improving institutions of 
higher education in South Carolina which are congruent with the 
accreditation standards and criteria of the Southern Association of 
Colleges and Schools* The first section defines key terms and states 
that overall institutional effectiveness reflects the quality, 
productivity, and efficiency of programs of instruction, scholarship, 
and service, as defined and prioritized in an institution's mission* 
The next section identifies 10 steps in assessing institutional 
effectiveness including: (1) defining program goals; (2) defining 
program objectives; (3) prioritizing objectives; (4). describing 
program strategies; (5) selecting assessment methods; (6) designing 
an assessment plan; (7) implementing the plan and measuring progress; 
(8) analyzing and evaluating results; (9) using results to identify 
program strengths and weaknesses; and (10) using evaluation 
information to improve programs and allocate resources* The following 
section offers recommended options for assessing academic programs 
and non-academic and administrative programs. The final section 
considers indicators of institutional effectiveness and lists 
potential quality indicators goruped into indicators of quality 
outcomes, quality processes, quality inputs, productivity/efficiency 
outcomes, process productivity/efficiency, and input 
product ivity/ef f iciency * (DB) 
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INTRODUCTION 



The purpose of this document is to provide definitions, guidelines, 
and recommended practices for assessing and improving institutional 
effectiveness in higher education. It is hoped that the information 
provided will help guide the development of assessment and institutional 
effectiveness policies and practices at the program, institution, and state 
levels to enhance the quality of education and services for all those 
served by higher education Institutions throughout South Carolina. 
These recommendations are also designed to be congruent with the 
accreditation standards and criteria of the Southern Association of 
Colleges and Schools. 
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DEFINING INSTITUTIONAL EFFECTIVENESS 
IN HIGHER EDUCA'nON 

Institutional effectiveness as a process in higher education is the 
close Integration of planning, budgeting, research, and assessment to 
achieve the mission of the Institution with quality, productivity and 
efftclency. Institutional effectiveness provides direction to the 
Institution, improves the quality of institutional programs and services, 
ai}d assures accountability to both internal and external constituencies 
by demonstrating the extent to which a college or university achieves its 
stated mission or purpose. 

Higher education Institutions generally have missions /stateme*' 
of purpose which include the primary goals of instruction, sdiolarshlp, and 
public service, although these will vary in emphasis from Institution to 
institution. 

• Instruction Includes aU curricular and support efforts designed 
to directly or indirectly result in student learning and 
development. This includes all academic programs, as well as 
academic support and student support programs. 

• Scholarship is broadly defined as contributions to the 
development, advancement, or dissemination of higher 
education or an academic discipline. It includes research and 
publication, artistic expression, professional practice, and other 
forms of scholarly production, as well as scholarship support 
efforts such as grant writing and professional development 



activities. 
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• Service consists of professional activities intended to benefit 
constituencies either inside or outside the institution. 

External service includes work for professional or disciplinaiy 
organizations, for other educational groups or institutions, for 
-• business -and industry, and for-individuals, groups, and 
agencies firom the local community to the society at large. 
Internal service includes, but is not limited to, administrative 
and committee work at all levels of the institution, extra- 
instructional work with students, and special projects or 
assignments. 

Each institution should also have systematic strategies to attain its 
goals. The functional strategic unit for institutional goal attainment is 
a program. Each program has mission-derived objectives serving one or 
more institutional goals, and personnel responsible for its operations. 

Program quality must be primarify determined by assessing results 
(outcomes, outputs) in relation to inputs and processes. To accomplish 
this, each program must operationally define its intended outcomes, and 
assess its own degree of success at outcomes attainment. Those 
assessed outcomes must then be evaluated in relation to Inputs (or 
starting points) and processes used, and compared to internal and 
external stsuidards of quality. Internal standards are those established 
by the program euid other institutional entities. External standards 
include “model" programs established by learned societies, or 
professional associations, “approved programs" promulgated by 
specialized accreditors, credentlallng organizations or governmental 
agencies, and equivalent programs at peer institutions. 
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Program productivity is the number or volume of successful 
outcomes or accomplishments. It is sometimes expressed as the ratio of 
program successes to failures. Program efficiency is the function of both 
productivity and cost (in money, time, and resources). In order to 
assess efficiency, each program must define the strategic resources - i.e., 
personnel, facilities, equipment and supplies - it utilizes to attain its 
outcomes, and their costs. Program productivify divided by program cost 
thus provides a measure of program efficiency which can be compared to 
similar internal and external standards as those for quality (e.g., 
institutional budget requirements, costs of “mcxlel” or equivalent 
programs on other campuses). 

Overall institutional effectiveness reflects ihe quality, productivity, and 
efficiency of programs of instruction, scholarship, and service, as defined 
and prioritized in an institution’s misslorL Quality must be determined 
independently from productivify and efficiency to avoid distorting the 
integrity of all, and qualify should be weighted more heavily in 
evaluating effectiveness. An institution’s effectiveness consists of the 
effectiveness of all its operations and programs. The prlmaiy purpose of 
assessment is to document and Improve such effectiveness. 
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ASSESSING INSTITUTIONAL EFFECTIVENBSS 
IN HIGHER EDUCATION 

For a higher education institution to assess its effectiveness, it 
must measure the quality, productivity, and efficiency of its programs 
and evaluate those results. Although an institution's programs will 
differ in many ways, the basic effectiveness assessment process for aU 
programs, academic and non-academic alike, will consist of common 
steps such as those outlined below. 

1. Define program goals, clearly related to (a) the mission and stated 
goals of the institution, (b) student/ service recipient interest, and 
(c) consumer needs. For academic programs, the goals of 
instruction, scholarship, and service should all be addressed, with 
instruction being the primary goal, students being the primary 
recipients, curriculum being the primary strategy, and more 
advanced educational programs, employers, or society-at-large being 
the primary consumers of program products or graduates. Most 
non-academic programs have a service priority, with students 
(directly or indirectly) or other internal or external constituencies 
being primary recipients and consumers. 

2. Define program objectives in operational or measurable terms that 
specify outcomes or what is expected from successful program 
completion. Instructional objectives should be stated as the student 
learning (knowledge and skills) and development (attitudes and 
behaviors) expected of those who successfully complete an academic 
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program. Scholarship £ind service objectives should also be stated In 
terms that define expected results or products. 

3. Prioritize objectives based on their Importance to mission and 
goals and on previously determined needs of service recipients and 
consumers. Such priorities should determine the relative emphasis 
on subsequent strategies and assessment methods for each 
objective. 

4. Describe program strategies responsible for assuring the 
attainment of program objectives and their cost. For academic 
programs, strategies would Include components of curriculum and 
Instruction and co-curricular requirements designed to produce or 
enhance student learning and development. Strategies also include 
the policies and procedures which foster and support scholarly and 
service outcomes. 

5. Select assessment methods which are expected to measure how 
well program strategies are attaining program objectives in a valid 
and practical manner. Valtdlty refers to the accuracy and utility of 
an assessment measure for the particular purpose it is being used to 
serve. Internal validity relates to how accurately the measure 
determines the attainment of program objectives. External validity 
relates to the generalization or comparability of results to other 
programs across the region or state. 

6. Design an assessment plan specifying how, when, where, and by 
whom program objectives will be assessed. It is extremely important 
for the plan to have the input of those knowledgeable about. 
Involved in, and affected by the attainment of objectives it is 
designed to assess. It should include multiple measures that meet 
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minimum criteria of internal validity, external validity, and 
practicality. Assessment methods should be implemented on a 
regulEir, systematic schedule designed to enhance the attainment of 
objectives, minimize disruptions to programs and services, and 
correspond with cycles of-planning, budgeting, and other decision 
making procedures. 

7. Implement plan/measure progress in attaining program 
objectives. The successful implementation of assessment methods 
requires administrative support and the cooperation of those 
Implementing and responding to such methods. Implementation 
should be carried out with the full population of persons or 
elements to be assessed or, where this is impossible or impractical, 
with a representative sample. Wherever possible, assessment should 
be embedded in ongoing activities (l.e., in course content or degree 
requirements, routine Acuity or department reporting, and service 
delivery and follow-up). 

8. Analyze and evaluate results. Results should be fully and 
accurately analyzed and evaluated by program personnel, 
supervisors, and assessment experts. Only results determined to be 
validly related to program quality, productivity, and/or efficiency 
should be used for subsequent decision-making and planning. 

9. Use valid results to determine program strengths and 
weaknesses. Results pertaining to program quality. pro<^uctlvity, 
and efficiency should be compared to internal and external 
standards of effectiveness In order to determine strengths, 
weaknesses, and needs for future Improvement. 
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10. Use resulting information to improve programs, determine 

planning and budgeting needs, allocate resources, and generate 
accountability reports. Such uses of assessment results constitute 
“closing the loop” In order to Improve the quality, productivity, 
efficiency,- and/or accountability of a program. Many instructional, 
curricular, policy, and/or prcxjedural Improvements can be 
accomplished relatively quickfy and forthrlghtty by program 
personnel making internal changes. Longer term and more costly 
changes may necessitate the support of the planning, budgeting, and 
resource allocation system of the institution or even the state or 
other funding bodies. The process of integrating assessment with 
decision-making should maximize quEillty and productivity while 
minimizing cost in the attainment of goals and objectives by an 
institution and its administration, faculty, staff, and students. 
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RECOMMENDED OPTIONS FOR ASSESSING PROGRAMS 
IN HIGHER EDUCATION 

Although no single measure can validly assess the range of 
objectives addressed by higher education programs, the following options 
for assessing academic, nonacademic, and administrative programs are 
recommended for consideration: 

Academic Program Assessment Measures 

• for student knowledge competencies - a localty-developed 
criterion-referenced measure such as a comprehensive exit 
examination (for internal validity), and a regionally or 
nationally normed standardized achievement test or one or 
more external examiner(s) from the program’s national 
association, or from a model or peer program at another 
institution (for external validity). Other examinations, 
criterion-referenced commercial tests, archival records, in-class 
probes or classroom research methods, portfolios, papers, or 
projects may also be useful in assessing student knowledge. 

• for student skills competencies - performance appraisal and/or 
performance simulations, plus archival records, portfolios, or 
other relevant performance-bsised projects deemed useful by 
program faculty. 

• for student attitude and behavior characteristics • behavioral 
observations, exit interviews, focus groups, and surveys of 
alumni and significant third parties, plus archival records, logs, 
and other portfolios. 

o 
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Non>Academic and Administrative Program Measures 

• Interviews with and/or surveys of at least a representative 
sample of service recipients and consumers, including open- 
-ended questions regarding their needs, the quantity and quality 
of services received, and suggestions for improving program 
objectives, strategies, and assessment measures. 

• objective measures (e.g., behavioral observations or archival 
records) of the quantity of services needed (e.g., number of 
service requests, by category) and quantity of services rendered 
(e.g., number of satisfactory services, by category). 

• whatever other information regarding the quality and/or 
quantity of services the program personnel deem useful. 
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INDICATORS OF INSTITUTIONAL EFFECTIVENESS 
IN HIGHER EDUCATION 



10 



The following section describes potential indicators of quality and 
productivity/ efficiency in higher education. No single indicator can be 
interpreted in Isolation or applied equally to every type of institution and 
program. Indicators should be viewed in the context of overall 
institutional effectiveness processes and in relation to 

A. the mission of the institution; 

B. the program goals and objectives that support the institution’s 
mission; and 

C. the abilities, characteristics, needs, and goals of incoming 
students and others served by the institution. 

Although indicators may provide a "snapshot” of some aspects of 
quality and efficiency, most do not typically answer the critical question 
of why a program or institution has performed at a particular level of 
effectiveness. A review of assessment data in general and indicators of 
quality and efficiency in particular is thus best accomplished within an 
oucmU joint review of effectiveness by experts both internal and external 
to the Institution. 

Each indicator listed below appears under one of two categories, 
quality or productivity /ejflclency , based on the aspect of institutional 
effectiveness primarily measured by that indicator. Additionally, three 
subcategories denote whether each indicator primarily measures: 
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• Inputs (I = what the institution receives to work with); 

• Processes (P = how the institution tries to accomplish its work, i. 
program strategies): 

• Outputs /outcomes/results (O = the products of its work); 

• Some combination of the three. 

(NOTE: Outcomes are the most important “bottom-line" 
indicators of effectiveness. The most valid indicators will describe 
outcomes /results and help identify the process strengths and/or 
weaknesses responsible for the results, so that Improvements can be 
made to enhance program and institutional effectiveness.) 
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POTENTIAL INDICATORS OF QUALITY IN HIGHER EDUCATION 
(Not listed in order of importance or practicality.) 

Indicators of Qtiality Outcomes /Results (O) [with Inputd) and 
Process(P) combinations noted in parentheses] 

• student achievement (O) and/or gains (O, I) in such areas as: 

A. Problem solving/creative thinking 

B. Critical thinking 

C. Scientific reasoning/ skills 

D. Aesthetic sensitivities /appreciation 

E. Quantitative skills 

F. Writing/communication skills 

G. Research skills 

H. Major studies 

I. Self-declared goals 

• Skilled performance in recitals, exhibits, internships or similar 
activities (O) 

• Student performance on a) licensing exams, b) professional exams, 
and c) graduate school admissions tests (O) 

• Employment rates of graduates in chosen field (for those students 
who express employment in chosen field as their primary post- 
graduation goal) (O) 

• Graduate school admission rates (for those students in senior 
institutions who express admission to graduate school as their 
primary post-graduation goal) (O) 

• Academic success of transfer students (for two-year institutions) (O) 
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• Academic success of undergraduate alumni in graduate programs (for 
senior institutions) (O) 

• Employer ratings of graduates (O) 

• Academic performance of targeted student groups (e.g., athletes, 
minorities, non-traditional students, etc.) (O) 

• Student evaluations of curriculum, instruction, and services (O, P) 

• Alumni ratings of the quantity and quality of preparation for cmeer or 
other post-graduation activities (O, P) 

• Alumni ratings of the quality of instruction, advising, and curriculum 
(O, P) 

• Special recognition /honors received by students, graduates, and 
alumni (O, P) 

• Ratings of the institution and its programs by external audiences 
(e.g., poUs, media lists, etc.) (O, P) 

• Ratio of nationally accredited programs to programs eligible for 
national accreditation (O, I, P) 

• External reviews of institution and programs (O, I, P) 

• Observed or reported alumni (or student) attitudes and behaviors 
(O, I, P) 

Indicators of guality Processes (P) (with Inputs(I) and Outcomes(O) 

combinations noted in parentheses] 

• Ratings of the institution and its programs by external evaluators (P) 

• Student ratings of instruction, advising, services, and programs (P) 

• Reports of program changes made as a result of internal or external 
reviews (P) 
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Faculty, staff, and student ratings of administrative organization, 
procedures, and personnel (P) 

Average class size (P) 

Equipment condition and amount (P) 

Lower-division cour-ses -taught by full-time or senior faculty (P) 
Percent of students receiving career advising (P) 

Percent of budget devoted to instruction and instructional support 
services (P) 

Percent of budget devoted to faculty and staff and f/s development (P) 
Personnel membership, participation, leadership roles, and/or offices 
held in professional associations and/or service organizations 
Types and range of student development programs offered (P) 

Student grievances (assuming similar grievance procedures exist for 
all institutions) (P) 

Student reports of out-of-class interactions with faculty (P) 

Ebctemal program reviews (P, O) 

Types and range of public service activities, and number/ types of 
persons receiving such services (P, O) 

Types, recognition, and number of faculty scholarly activities (P, O) 
Special recognition/honors received by faculty and staff (P, O) 
Alumni ratings of instruction, advising, services and programs (P, O) 
Faculty, staff, and administrative ratings of advising, services, and 
programs (P, O) 

Percent of faculty with terminal degrees in appropriate field (P, 1) 
Ratio of nationally accredited programs to programs eligible for 
national accreditation (P, 1, O) 
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Indicators of Quality Inputs [with Process(P) combinations noted in 

parentheses] 

• SAT scores and other admission or placement test results, and high 
school ranks of incoming students (I) 

• Percent of studentsmeeting full admission requirements and 
prerequisites (I) 

• Incoming student collegiate and career goals (I) 

• Percentage of admitted students requiring developmental/remedial 
courses (I, P) 

• First term performance of transfers (I, P) 
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INDICATORS OF INSTITUTIONAL PRODUCTIVITY/EFFICIENCy 

Indicators of Productivity/EfSciency Outcomes/Results [with 

Process (P) combinations noted in parentheses] 

• Graduation rates (4 yr. /6 yr. for senior institutions, 2 yr. /3 yr. for 
two-year institutions) (O) 

• Graduate placement rates (O) 

• Number/ percent of targeted student groups (e.g., minorities, non- 
traditlonals, athletes, particular skills, etc.) graduating (O) 

• Numbers of persons served through continuing education offerings (O) 

• Number of faculty publications, performances, presentations to 
professional groups, and similar scholarly activities (O, P) 

• Number and types of public service /community outreach activities 
and number and types of persons benefiting from such activities (O, P) 

• Number /percent of targeted faculty or staff emplo 3 mient (e.g., 
minorities, women, etc.) (O, P) 

Indicators of Process Productivity/Efficiency [with Output(0) and 

Inputd) combinations noted in parentheses] 

• Credit hour production (P) 

• Costs per FTE (P) 

• Retention and attrition rates (P) 

Indicators of Input Productivity/Efflciency [with Input (1) 

combinations noted in parentheses] 

• Institutional income (i.e., budget allocation, grants, gifts, etc.) (I) 

• Payroll (I) 

• Expenditures per FTE (P, I) 
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